Image processing apparatus

ABSTRACT

An image processing apparatus for processing RGB image data output from an image capturing element including a primary-color filter, includes: a middle-high range luminance component compensation section for compensating for a middle-high range luminance component of a low-frequency luminance signal generated based on the RGB image data such that the low-frequency luminance signal has substantially an ideal frequency luminance characteristic which is lower than or equal to a predetermined frequency.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to an image processing apparatusfor processing image data obtained by a CCD (charge-coupled device) areasensor having a primary-color filter so as to obtain a high qualityimage. Such an image processing apparatus is mounted on a digital camera(e.g., electronic still camera) or the like.

[0003] 2. Description of the Related Art

[0004] A conventional image processing apparatus for use in a digitalcamera, or the like, performs a series of various image processingsteps, e.g., interpolation for each color component, contour emphasizingprocessing, etc., on image data obtained by a CCD area sensor having aprimary-color filter, so as to obtain a high quality image. Hereinafter,conventional image processing apparatuses are described with referenceto FIGS. 13 through 21.

[0005]FIG. 13 is a block diagram showing a first exemplary structure ofa conventional image processing apparatus. In FIG. 13, the imageprocessing apparatus 100 includes an optical low-pass filter 101, aprimary-color CCD area sensor 102, a RGB interpolation section 103, awhite balance adjustment section 104, a gamma correction section 105,and a sharpening processing section (contour emphasizing section) 106.

[0006] The low-pass filter 101 removes frequency components whosefrequency is equal to or higher than a ½ of sampling frequency fs.According to the sampling theorem, when an image is converted into imagedata, frequency components whose frequency is equal to or higher than a½ of sampling frequency fs are converted into aliasing noise. In orderto reduce such aliasing noise, the optical low-pass filter 101 isprovided before the CCD area sensor 102.

[0007] The primary-color CCD area sensor 102 is formed by a plurality oflight-receiving elements arranged in a matrix. The primary-color CCDarea sensor 102 includes a primary-color (R, G, B) filter on alight-receiving face thereof where each of color areas corresponds toone pixel. The color filter is formed based on the Bayer array (FIG. 14)which is a type of RGB arrangements.

[0008] The RGB interpolation section 103 (which will be described laterin detail) can receive, from the CCD area sensor 102, only one type ofcolor component for each pixel among R, G, B colors of the color filterof the sensor 102. In the RGB interpolation section 103, each of theremaining two types of color components in each pixel is calculated fromcolor components of the same type in neighboring pixels, whereby all ofthe three color components are obtained in each pixel.

[0009] The white balance adjustment section 104 adjusts the whitebalance with respect to the R-, G-, and B-components in each pixel whichare obtained by the RGB interpolation section 103 according to the colortemperature of light so as to correct the color of an image.

[0010] The gamma correction section 105 processes the R-, G-, andB-components obtained after the white balance has been adjusted suchthat the R-, G-, and B-components conform to properties of a display orprinter which outputs an image.

[0011] The sharpening processing section 106 performs contouremphasizing processing (sharpening processing) in order to obtain asharp image. The sharpening process compensates for high-range luminancecomponents which have been removed by the low-pass filter 101 byfiltering processing which emphasizes high-range luminance components.That is, in the case where the low-pass filter 101 is provided, CCDimage data generated by the primary-color CCD area sensor 102 has areduced high-range luminance component. Furthermore, for example, sincean unobtained color component in each pixel is compensated for byinterpolation, e.g., by calculating an average value from colorcomponents of the same type in neighboring pixels, an image obtained bysuch interpolation has a further reduced high-range luminance component.Thus, in order to obtain a sharp image, it is necessary to compensatefor a lost high-range luminance component by performing a filteringprocess which emphasizes high-range luminance components.

[0012] Interpolation by the RGB interpolation section 103 is nowdescribed in detail.

[0013] When the CCD area sensor 102 is a single-plate area sensor, inthe RGB interpolation section 103, interpolation is performed for themissing two types of color components such that all of R(red)-,G(green)-, and B(blue)-color components are complete in each pixel.Based on all of R-, G-, and B- color components including colorcomponents obtained by interpolation, a color image is created.

[0014] When the CCD area sensor 102 is a three-plate area sensor wherethree area sensor elements are provided with full-size R-, G-, andB-color filters, respectively, the RGB interpolation section 103receives from the CCD area sensor 102 all of R-, G-, and B-colorcomponents for each pixel. Thus, it is not necessary to performinterpolation. However, when the CCD area sensor 102 is a single-platearea sensor having, on its light-receiving element, a single colorfilter on which R-, G-, and B-filter areas are arranged in apredetermined pattern (e.g., Bayer array shown in FIG. 14), it isnecessary to perform interpolation for missing color components in eachpixel. Therefore, when employing a single-plate CCD camera incorporatinga single-plate CCD area sensor, missing color components in each pixelwhich cannot be obtained by a color filter are created by variousmethods.

[0015] U.S. Pat. Nos. 4,605,956, 4,642,678, and 4,630,307, and thedocument by James E. Adams, Jr., “Interactions between color planeinterpolation and other image processing functions in electronicphotography”, suggest various interpolation methods for the purpose ofcreating an image with no jaggy or zip noise. “Jaggy” means astep-shaped noise which emerges along a contour of an image, which is atype of noise among various types of noise.

[0016] In U.S. Pat. No. 4,605,956, interpolation is performed by usingthe following expressions (1) through (4) on missing color components ina pixel arrangement shown in FIG. 15 (bilinear interpolation method).Interpolation for G- and B-components at each pixel position in FIGS. 14and 15 is shown below:

G 5=(G 2+G 4+G 6+G 8)/4  (1)

B 5=(B 1+B 3+B7+B 9)/4  (2)

B 2=(B 1+B 3)/2  (3)

B 4=(B 1+B 7)/2  (4)

[0017] In the Bayer array shown in FIG. 15, R-, G-, and B-colors arearranged according to a certain pattern. For example, at a central pixelposition, R5, color components G5 and B5 are missing in a correspondingcolor filter. The color components G5 and B5 are obtained fromexpressions (1) and (2).

[0018] Furthermore, B2 is obtained from expression (3), and B4 isobtained from expression (4). Interpolation for each of B6 and B8 isperformed by obtaining an average value of B-component values invertically or horizontally neighboring pixels based on an expressionsimilar to expression (3) or (4). This is the same for R2, R4, R6, andR8.

[0019] Furthermore, G1 is obtained from G-component information inneighboring pixels around pixel position “B1” by an expression similarto expression (1). This is the same for G3, G7, and G9. R1 is obtainedfrom R-component information in neighboring pixels around pixel position“B1” by an expression similar to expression (2). This is the same forR3, R7, and R9.

[0020] Next, an image processing method disclosed in Japanese Laid-OpenPublication No. 10-164371 is described with reference to FIG. 16.

[0021]FIG. 16 is a block diagram showing a second exemplary structure ofa conventional image processing apparatus. As shown in FIG. 16, theimage processing apparatus 200 includes a primary-color CCD area sensor201, a RGB interpolation section 202, and a contour emphasizing sections203 provided for respective ones of R-, G-, and B-components.

[0022] The RGB interpolation section 202 first interpolates aG-component by using expression (5) which represents median processingfor G-color components obtained by the filter arrangement shown in FIG.15. Then, only at pixel positions, “R” and “B”, in FIG. 14, a (R-G)component and a (B-G) component are created. In the last, the created(R-G) and (B-G) components are interpolated by bilinear interpolation,and G-components are added to the interpolated (R-G) and (B-G)components, whereby R- and B-components are obtained.

G 5=(G 2+G 4+G 6+G 8−Min−Max)/2  (5)

Min=Min(G 2,G 4,G 6,G 8)

Max=Max(G 2,G 4,G 6,G 8)

[0023] How to obtain B4 at pixel position “G4” is now described indetail with reference to FIG. 15. In the first step, G1, G3, G5, G7, andG9 are interpolated by the median processing process using expression(5). Then, (B1-G1) and (B7-G7) which are (B-G) components at pixelpositions “B1” and “B7” are created. (B4-G4) which is a (B-G) componentat pixel positions “G4” is represented as follows:

B 4−G 4=(½){(B 1−G 1)+(B 7−G 7)}

[0024] From this expression, B4 can be represented as follows:

B 4=(½){(B 1−G 1)+(B 7−G 7)}+G4

[0025] The contour emphasizing sections 203 employs a two-dimensionalsecond-derivative filter shown in FIG. 17A for each of R-, G-, andB-components. Each box of the two-dimensional second-derivative filtershown in FIG. 17A corresponds to one pixel, and the number shown in eachbox represents a weight. The weight of each box area is set such thatthe total weight in the two-dimensional second-derivative filter iszero.

[0026] Next, an interpolation method disclosed in Japanese Laid-OpenPublication No. 11-18047 is described with reference to FIG. 18.

[0027]FIG. 18 is a block diagram showing a third exemplary structure ofa conventional image processing apparatus. In FIG. 18, the imageprocessing apparatus 300 includes a primary-color CCD area sensor 301,RGB interpolation sections 302, a middle-range component emphasizingsection 303, a high-range component emphasizing section 304, a whitebalance adjustment section 305, and a gamma correction section 306. Theprimary-color CCD area sensor 301 includes a color filter of Bayerarray.

[0028] The RGB interpolation sections 302 perform interpolation on eachof R-, G-, and B-components. When a G-component is interpolated by usinga median method of expression (5), and R- and B-components areinterpolated by using a bilinear method of expressions (2) through (4),interpolation is achieved with high image quality.

[0029] The middle-range component emphasizing section 303 is formed by amiddle-range component extraction section 303 a and adders 303 b. In themiddle-range component emphasizing section 303, the middle-rangecomponent extraction section 303 a extracts a middle-range componentfrom a G-component interpolated by the G-interpolation section of theRGB interpolation sections 302. The extracted middle-range component issynthesized with each of R-, G-, and B-components by the adders 303 b,whereby the middle-range component emphasizing processing is achieved.In this middle-range component emphasizing processing, the differencebetween a G-component and a low-pass filtered G-component is added as acompensation component to the G-component. In other words, a high-rangecomponent of the G-component is removed by a low-pass filter (not shown)to obtain a GBAR-component, and a compensation component which is thedifference between the G-component and the GBAR-component (G-GBAR) isadded to the G-component, whereby a middle-range component emphasizingprocessing is achieved.

[0030] The high-range component emphasizing section 304 is formed by ahigh-range component extraction section 304 a and adders 304 b. In thehigh-range component emphasizing section 304, the high-range componentextraction section 304 a extracts a high-range component from aG-component interpolated by the G-interpolation section of the RGBinterpolation sections 302. The extracted high-range component issynthesized with each of R-, G-, and B-components by the adders 304 b,whereby high-range component emphasizing processing is achieved. Thishigh-range component emphasizing process employs the two-dimensionalsecond-derivative filter shown in FIG. 17B. In the filter shown in FIG.17B, in order to obtain pixel data of a pixel corresponding to a centralbox “4”, the weight of the filter is multiplied by data from each pixel,and the products for all of the pixels are added up. Thus, when a resultof this calculation does not change, the sum of the products results inzero. When a result of this calculation changes much, the sum becomes alarge value, whereby a high-range component is emphasized.

[0031] Now, consider a case where each of R-, G-, and B-componentsobtained through a Bayer array filter shown in FIG. 14 is sampled at asampling frequency fs=1/Δx=1/Δy. A sampling frequency distribution rangeobtained in this case is shown in FIG. 19. Herein, “Δx” denotes thewidth of a pixel (pixel pitch) in a horizontal direction (x-direction),and “Δy” denotes the width of a pixel (pixel pitch) in a verticaldirection (y-direction).

[0032] According to the sampling theorem, a highest restorable frequencywithin a spatial frequency band of an original image is represented by asolid-line lozenge which is indicated by an arrow “G-component” in FIG.19. Furthermore, R- and B-components are included in a two-dot chainline square. Thus, a frequency band which can be accurately convertedinto R-, G-, and B-components of image data is an area within the squaredefined by the two-dot chain line. As seen from FIG. 19, theaccurately-restorable, highest frequency of the original image is a halfof the sampling frequency fs(=1/Δx=1/Δy).

[0033] Therefore, a frequency component which is higher than the highestrestorable frequency fs/2 emerges as a noise in the image data.

[0034] In general, for the purpose of avoiding such a problem, anoptical low-pass filter (anti-aliasing filter) is provided to a CCD areasensor. This low-pass filter removes frequency components of image datawhich are higher than the highest restorable frequency fs/2, butundesirably attenuates some of frequency components which are lower thanthe highest restorable frequency fs/2 because the low-pass filter is notideal. In FIG. 20, graph a shows an ideal low-pass filter frequencycharacteristic (i.e., only frequency components which are higher thanthe highest restorable frequency fs/2 are removed). However, in anactual case, the low-pass filter frequency characteristic results in acurve of graph b. Furthermore, graph a shows an ideal frequencycharacteristic of a compensation filter which is used to compensate forthe characteristic of graph b so as to obtain a characteristic similarto the ideal characteristic of graph a. According to the presentinvention, as described later in detail, a middle-range luminancecomponent and a high-range luminance component are synthetically addedto a newly-extracted, middle-high range luminance component according toa predetermined synthetic ratio, whereby a low-pass filtered componentis compensated for by using a compensation filter having a frequencycharacteristic similar to graph c.

[0035] Furthermore, when interpolation is performed on components of animage which have been obtained through a Bayer array filter shown inFIG. 14 so as to compensate for the missing two types of colorcomponents in each pixel, high-range luminance components areattenuated. Accordingly, it is indispensable for generating a sharpimage to compensate for attenuated high-range components. In general,such a compensation is achieved by separately performing a compensationfor middle-range luminance components with a compensation filter havinga frequency characteristic of graph d and a compensation for high-rangeluminance components with a compensation filter having a frequencycharacteristic of graph c. In FIG. 21, graph b shows a syntheticfrequency characteristic obtained after low-pass filtering (with ananti-aliasing filter) and interpolation processing, and graph a shows anideal frequency characteristic obtained after an entire process in animage processing system including compensation processing.

[0036] In the above-described conventional art, attenuated middle- andhigh-range components among low-pass filtered components are compensatedfor by compensation filters. The middle-range components are compensatedfor by a compensation filter having a characteristic represented bygraph d of FIG. 21, and the high-range components are compensated for bya compensation filter having a characteristic represented by graph a ofFIG. 21. When graph d exhibits the maximum amplitude, the angularfrequency ω is π/2 (equivalent to fs/4). When graph a exhibits themaximum amplitude, the angular frequency ω is π (equivalent to fs/2).

[0037] However, in contour emphasizing processing which is performed forobtaining a sharp image, when high-range components of an image areemphasized by compensating for high-range components, all componentshaving an angular frequency ω higher than fs/2 are converted into noise.Thus, as the resolution of the image is increased, noise and jaggy areobserved more prominently.

[0038] For example, in the method described in connection with the firstexemplary conventional structure shown in FIG. 13, a G-component whichlargely contributes to the luminance of an image is interpolatedaccording to above expression (1), and image data obtained after theinterpolation is subjected to contour emphasizing processing, and as aresult, jaggy emerges in an edge portion of the image. Examples ofproposed method for solving such a problem include a color-smoothinginterpolation method (U.S. Pat. No. 4,642,678), a pattern-recognitioninterpolation method (U.S. Pat. No. 4,630,307), an adaptiveinterpolation method (document by James E. Adams, Jr.). These methodsare different interpolation methods, but all performed in a structuresimilar to the image processing apparatus 100 shown in FIG. 13. That is,a high-range component which is indispensable for obtaining a sharpimage is generated by contour emphasizing processing in the sharpeningprocessing section 106 which resides at the end of the entire process.These interpolation methods each have some advantages in reducing falsecolor and jaggy. However, these interpolation methods do not have meansfor processing a high-range component except for high-range emphasizingfiltering, and thus, noise occurs in the high-range emphasizingfiltering.

[0039] Thus, in the methods described in connection with the above firstthrough third exemplary conventional structures (FIGS. 13, 16, and 18),noise is inevitably emphasized when a high-range component iscompensated for by sharpening processing. That is, during interpolationof high-range components, frequency components whose frequency is equalto or higher than ½ of a sampling frequency fs are also emphasized, andthe emphasized components whose frequency is equal to or higher than a ½of sampling frequency fs are observed as noise and/or jaggy.

SUMMARY OF THE INVENTION

[0040] According to one aspect of the present invention, an imageprocessing apparatus for processing RGB image data output from an imagecapturing element including a primary-color filter includes: amiddle-high range luminance component compensation section forcompensating for a middle-high range luminance component of alow-frequency luminance signal generated based on the RGB image datasuch that the low-frequency luminance signal has substantially an idealfrequency luminance characteristic which is lower than or equal to apredetermined frequency.

[0041] In this specification, a “middle-high range luminance component”means a luminance component mainly containing middle-high rangecomponents, and a “low-frequency luminance signal” means a luminancesignal mainly containing low-frequency components.

[0042] With the above structure, a middle-high range luminance componentof a low-frequency luminance signal which is attenuated as compared withan ideal frequency characteristic in a range of a predeterminedfrequency (a ½ of sampling frequency fs) or smaller is compensated.Thus, a sharp image can be obtained by contour emphasizing processingwhile preventing occurrence of noise and jaggy which may be caused whenobtaining the sharp image.

[0043] According to another aspect of the present invention, an imageprocessing apparatus for processing RGB image data output from an imagecapturing element including a primary-color filter includes: amiddle-high range luminance component extraction section for extractinga middle-high range luminance component which has a zero amplitude at anangular frequency ω=π and a maximum amplitude at an angular frequency ωbetween π/2 and π from a first luminance signal generated based on RGBimage data, and a first synthesis section for adding the middle-highrange luminance component to a low-frequency luminance signal generatedbased on the RGB image data so as to generate a second luminance signal.

[0044] With such a structure, an image with high resolution can beobtained by contour emphasizing processing while preventing occurrenceof noise and jaggy which may be caused when obtaining the sharp image.

[0045] In one embodiment of the present invention, the middle-high rangeluminance component extraction section uses at least one filter having asize of an even-number of pixels to arithmetically process the firstluminance signal.

[0046] With such a structure, a middle-high range component whoseamplitude is zero when the angular frequency ω is π and has a maximumvalue at a position where the angular frequency ω is between π/2 to πcan be readily extracted.

[0047] In another embodiment of the present invention, the filter havinga size of an even-number of pixels is a two-dimensional filter and hascoefficients symmetrically arranged with respect to a x-direction and ay-direction.

[0048] With such a structure, a uniform filtering effect can beobtained, and as a result, an image can be faithfully reproduced.

[0049] In still another embodiment of the present invention, the filterhaving a size of an even-number of pixels includes a first low-passfilter having a differentiation capability and a second low-pass filter;and a difference between an output obtained by arithmetically processingthe first luminance signal using the first low-pass filter and an outputobtained by arithmetically processing the first luminance signal usingthe second low-pass filter is output as the middle-high range luminancecomponent.

[0050] With such a structure, arithmetic operations in x- andy-directions of image data can be separately performed, and accordingly,increases in the amount of arithmetic operations can be suppressed.Thus, YH extraction filtering can be readily implemented by hardware.

[0051] In still another embodiment of the present invention, the imageprocessing apparatus further includes: a first interpolation section forinterpolating missing components among R-, G-, and B-components for eachpixel before the generation of the first luminance signal, wherein thefirst interpolation section interpolates missing components byarithmetically processing the RGB image data using a filter having asize of 3 pixels×3 pixels.

[0052] With such a structure, a middle-high range luminance componentcan be extracted while most-effectively preventing deterioration of themiddle-high range luminance component.

[0053] In still another embodiment of the present invention, the imageprocessing apparatus further includes: a second interpolation sectionfor interpolating missing components among R-, G-, and B-components foreach pixel before the generation of the low-frequency luminance signal,wherein the second interpolation section interpolates missing componentsby arithmetically processing the RGB image data using a filter having asize of an even-number of pixels.

[0054] With such a structure, when compensating for a middle-high rangeluminance component of a low-frequency luminance signal, a center of themiddle-high range luminance component is present at a position betweenneighboring pixels, and a center of the low-frequency luminance signalis also present at a position between neighboring pixels. Thus,occurrence of a ghost in a reproduced image can be prevented.

[0055] In still another embodiment of the present invention, at leastone of the first and second interpolation sections interpolates the RGBimage data by using a median method for a G-component and a bilinearmethod for R- and B-components.

[0056] With such a structure, a G-component is interpolated by using amedian method, whereby attenuation of a high-range luminance componentis suppressed to a minimum level. R- and B-components are interpolatedby using a bilinear method, whereby noise is reduced. Thus, a contour ofan image is emphasized, and the quality of the image can be improved.

[0057] In still another embodiment of the present invention, the imageprocessing apparatus further includes: a middle/high-range luminancecomponent extraction section for extracting at least one of amiddle-range luminance component and a high-range luminance componentbased on the second luminance signal; and a second synthesis section foradding at least one of the middle-range luminance component and thehigh-range luminance component to the second luminance signal so as togenerate a third luminance signal.

[0058] In this specification, a “middle-range luminance component” meansa luminance component mainly containing middle-frequency components, anda “high-range luminance component” means a luminance component mainlycontaining high-frequency components.

[0059] With the above structure, by changing a ratio between amiddle-range luminance component and a high-range luminance component,the three-dimensional appearance (stereoscopic effect or stereophoniceffect) of an image can be adjusted according to user's preference.

[0060] In still another embodiment of the present invention, wherein themiddle/high-range luminance component extraction section arithmeticallyprocesses the second luminance signal by using one filter which has anadjustable coefficient.

[0061] With such a structure, a middle/high-range component extractionsection can be readily formed of a single filter.

[0062] In still another embodiment of the present invention, the imageprocessing apparatus further includes: a median filtering section forremoving, with a median filter, noise inherent to the image capturingelement which is contained in a color-difference signal generated basedon a RGB image signal from the second interpolation section, wherein themedian filtering section changes the size of the median filter accordingto an amount of the noise.

[0063] With such a structure, the amount of noise included in acolor-difference signal varies according to, for example, the quality ofan image capturing element such as a CCD. Thus, by selecting anappropriate median filter according to the amount of noise, acolor-difference signal with reduced noise can be generated.

[0064] Thus, the invention described herein makes possible theadvantages of providing an image processing apparatus which can preventnoise and jaggy which may occur when obtaining a sharp image.

[0065] These and other advantages of the present invention will becomeapparent to those skilled in the art upon reading and understanding thefollowing detailed description with reference to the accompanyingfigures.

BRIEF DESCRIPTION OF THE DRAWINGS

[0066]FIG. 1 is a block diagram showing a structure of an imageprocessing apparatus according to an embodiment of the presentinvention.

[0067]FIGS. 2A and 2B show specific examples of YH extraction filtersused in a middle-high range luminance component extraction section ofthe image processing apparatus shown in FIG. 1.

[0068]FIG. 3 shows frequency characteristics of the YH extractionfilters shown in FIGS. 2A and 2B and a frequency characteristic of adifferential output of the YH extraction filters shown in FIGS. 2A and2B.

[0069]FIG. 4 shows a specific example of a composite contour emphasizingfilter used in a middle/high range luminance component extractionsection of the image processing apparatus shown in FIG. 1.

[0070]FIGS. 5A and 5B show RGB interpolation filters used in a secondRGB interpolation section of the image processing apparatus shown inFIG. 1. FIG. 5A shows a specific example of an R- and B-interpolationfilter. FIG. 5B shows a specific example of a G-interpolation filter.

[0071]FIG. 6 shows examples of one-dimensional filters in parts (A)through (G).

[0072]FIG. 7 shows frequency characteristics of filter A having a sizeof an even-number of pixels and filters B and C each having a size of anodd-number of pixels.

[0073]FIG. 8 shows frequency characteristics of filter A having a sizeof an even-number of pixels and filters D and E each having a size of anodd-number of pixels.

[0074]FIG. 9 shows frequency characteristics of various filters eachhaving a different size of an even-number of pixels.

[0075]FIG. 10 illustrates a case where pixel data is generated betweenneighboring pixels with a filter having a size of an even-number ofpixels.

[0076]FIG. 11 shows frequency characteristics of a low-frequencyluminance signal, a middle-high luminance component, and a low-frequencyluminance signal including a compensated middle-high luminancecomponent.

[0077]FIG. 12 shows frequency characteristics of a middle-high rangeluminance component extraction filter, a middle-range luminancecomponent extraction filter, and a high-range luminance componentextraction filter.

[0078]FIG. 13 is a block diagram showing a first exemplary structure ofa conventional image processing apparatus.

[0079]FIG. 14 is a plan view showing a Bayer-array of a color filter.

[0080]FIG. 15 shows a portion of the Bayer-array of FIG. 14.

[0081]FIG. 16 is a block diagram showing a second exemplary structure ofa conventional image processing apparatus.

[0082]FIGS. 17A and 17B show two-dimensional second-derivative filterswhich are used in contour emphasizing processing.

[0083]FIG. 18 is a block diagram showing a third exemplary structure ofa conventional image processing apparatus.

[0084]FIG. 19 shows a restorable range of sampling frequencies for eachcolor component in the Bayer array.

[0085]FIG. 20 shows frequency characteristics of an optical low-passfilter and a compensation filter.

[0086]FIG. 21 shows frequency characteristics of a middle-rangeluminance component, a middle-high range luminance component, and ahigh-range luminance component compensation filter.

[0087]FIG. 22 shows a specific example of a YH extraction filter formedof a single filter which is used in a middle-high luminance componentextraction section of the image processing apparatus shown in FIG. 1.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0088] Hereinafter, an embodiment of the present invention will bedescribed with reference to the drawings.

[0089]FIG. 1 is a block diagram showing a structure of an imageprocessing apparatus according to an embodiment of the presentinvention. In FIG. 1, the image processing apparatus 1 includes: anoptical low-pass filter 2; a primary-color CCD area sensor 3; a firstRGB interpolation section (RGB interpolation section for extraction ofmiddle-high range luminance component) 4; a luminance generation section5 for extraction of middle-high range luminance component; a middle-highrange luminance component extraction section 6; a multiplier 7 and adder8 which function as a first synthesizing section; a middle/high rangeluminance component extraction section 9; and a multiplier 10 and adder11 which function as a second synthesizing section. The first RGBinterpolation section 4, the luminance generation section 5, and themiddle-high range luminance component extraction section 6 form amiddle-high range luminance component compensation section 20.

[0090] The low-pass filter 2 removes frequency components whosefrequency is equal to or higher than a ½ of sampling frequency fs.

[0091] The primary-color CCD area sensor 3 is formed by a plurality oflight-receiving elements arranged in a matrix. The primary-color CCDarea sensor 3 includes a color filter on a light-receiving face thereof.This color filter is formed based on the Bayer array (FIG. 14). Itshould be noted that RGB image data obtained from the primary-color CCDarea sensor 3 successively goes through a Correlated Double Sampling(CDS) circuit (not shown) which reduces noise in the RGB image data, anAutomatic Gain Control (AGC) circuit (not shown) for performing gainadjustment processing, and an A/D conversion circuit (not shown) whoseresolution is, e.g., 10 bits, and then reaches, as digital image data,the first RGB interpolation section 4 and a second RGB interpolationsection (RGB interpolation section for generation of low-frequencyluminance signal generation and color-difference signal) 12. (The secondRGB interpolation section 12 will be described later in detail.)

[0092] In the first RGB interpolation section 4, missing colorcomponents among R-, G-, and B-components for each pixel areinterpolated based on data from color components of the same type inneighboring pixels. In this example, the R- and B-components areinterpolated by a bilinear method represented by expressions (2) through(4), and the G-component is interpolated by a median method representedby expressions (5). The first RGB interpolation section 4 uses, as anRGB interpolation filter, a two-dimensional filter having a size of (anodd-number of pixels)×(an odd-number of pixels) which is greater than 3pixels×3 pixels.

[0093] As the size of this interpolation filter becomes larger, a largeramount of middle- and high-luminance components are reduced by thisfilter. As a result, it becomes more difficult to extract the middle-and high-luminance components. Interpolation processing would not beachieved by a filter of one-pixel by one-pixel. Furthermore, if atwo-dimensional filter of 2 pixels×2 pixels is employed, a bilinearinterpolation method cannot be used, and therefore, interpolation cannotbe achieved without using a nearest neighbor interpolation. When thenearest neighbor interpolation is used, middle-and high-luminancecomponents are deteriorated as compared with a case where a bilinearinterpolation method is used. Thus, a two-dimensional filter of 3pixels×3 pixels is desirable.

[0094] The luminance generation section 5 uses a first RGB image signalwhich is obtained by the first RGB interpolation section 4 so as togenerate a luminance signal Y for extracting middle- and high-luminancecomponents:

Y=0.30R+0.59G+0.11B  (6)

[0095] In the middle-high range luminance component extraction section6, the luminance signal Y which is generated by the luminance generationsection 5 is subjected to a first YH extraction filter F1 shown in FIG.2A and a second YH extraction filter F2 shown in FIG. 2B, wherebymiddle-high range luminance component YH is extracted. Specifically, themiddle-high range luminance component extraction section 6 outputs adifference between an output from the first YH extraction filter F1having a size of 6 pixels×6 pixels shown in FIG. 2A and an output fromthe second YH extraction filter F2 having a size of 4 pixels×4 pixelsshown in FIG. 2B as middle-high range luminance component YH. In the YHextraction filters 1 and 2, x denotes an operation in a horizontaldirection, and y denotes an operation in a vertical direction.

[0096] The first YH extraction filter F1 shown in FIG. 2A is a low-pathfilter having a differential effect (which includes positive andnegative coefficients). The first YH extraction filter F1 is symmetricwith respect to x- and y-directions, and may have a size of aneven-number of pixels along both x- and y-directions (4 pixels×4 pixels,8 pixels×8 pixels, 10 pixels×10 pixels, etc.). Furthermore, it isdesirable that the first YH extraction filter F1 has a square shape(having a size of a same number of pixels along both x- andy-directions). However, the first YH extraction filter F1 may have ahorizontally-oblong, rectangular shape where the size in the x-directionis larger than the size in the y-direction if the x-direction of animage is emphasized. Alternatively, the first YH extraction filter F1may have a vertically-oblong, rectangular shape where the size in they-direction is larger than the size in the x-direction if they-direction of an image is emphasized. Furthermore, in the first YHextraction filter F1, coefficients of terms are symmetrically arrangedwith respect to both x- and y-directions of the filter F1.

[0097] The second YH extraction filter F2 shown in FIG. 2B is a low-pathfilter (which includes only positive coefficients). The second YHextraction filter F2 has a size of an even-number of pixels along bothx- and y-directions (but smaller than the size of the filter F1).Furthermore, it is desirable that the second YH extraction filter F2 hasa square shape. However, the second YH extraction filter F2 may have ahorizontally-oblong, rectangular shape or a vertically-oblong,rectangular shape. Furthermore, in the second YH extraction filter F2,coefficients of terms are symmetrically arranged with respect to both x-and y-directions of the filter F2. With such a symmetrical arrangementof coefficients, an effect of filtering can be uniformly obtained, andas a result, an image can be faithfully reproduced.

[0098] With such a YH extraction filter formed by two filters,operations in x- and y-directions can be separately performed. Thus, YHextraction filtering can be readily implemented by hardware. FIG. 3shows frequency characteristics of the YH extraction filters F1 and F2(FIGS. 2A and 2B) and a frequency characteristic of a differentialoutput of the filters F1 and F2. In FIG. 3, graph h shows a frequencycharacteristic of the first YH extraction filter F1, graph i shows afrequency characteristic of the second YH extraction filter F2, andgraph j shows a frequency characteristic of middle-high range luminancecomponent YH which is the differential output of graph i and graph j. Inthis example, the YH extraction filter is formed by two filters, but maybe formed by a single filter. In such a case, the arrangement ofcoefficients of the filter is as shown in FIG. 22, and the amount ofarithmetic operations is increased.

[0099] The multiplier 7 gives supplemental compensation, by multiplyinga gain α which is an adjustment coefficient, to middle-high rangeluminance component YH which has been extracted by the middle-high rangeluminance component extraction section 6.

[0100] The adder 8 adds a middle-high range luminance componentmultiplied by a predetermined coefficient of gain α, αYH, to alow-frequency luminance signal YL which is obtained by a low-frequencyluminance signal generation section 15 (described later), therebygenerating a luminance signal (YL+αYH).

[0101] In the middle/high-range luminance component extraction section9, a luminance signal (YL+αYH) received from the adder 8 is subjected toa composite contour emphasizing filter shown in FIG. 4, wherebymiddle/high-range luminance components ENH are extracted. In the filterof FIG. 4, the ratio of extracted middle-range luminance components andextracted high-range luminance components can be adjusted by adjustingthe ratio between variables 1 and m.

[0102] The multiplier 10 gives supplemental compensation, by multiplyinga gain β which is an adjustment coefficient, to middle- and high-rangeluminance components ENH which has been extracted by themiddle/high-range luminance component extraction section 9.

[0103] The adder 11 adds middle- and high-range luminance componentsmultiplied by the multiplier 10 by a predetermined coefficient of gainβ, βENH, to a luminance signal (YL+αYH) which is obtained from the adder8, and outputs a luminance signal (YL+αYH+βENH).

[0104] The image processing apparatus 1 further includes: a second RGBinterpolation section 12 connected to the primary-color CCD area sensor3; a white balance adjustment section 13; a gamma correction section 14;a low-frequency luminance signal generation section 15; acolor-difference signal generation section 16; and a median filter 17.

[0105] In the second RGB interpolation section 12, interpolation isachieved by arithmetic operations using interpolation filters shown inFIGS. 5A and 5B. Color components interpolated by the second RGBinterpolation section 12 and color components obtained by the colorfilter are combined so as to generate a second RGB image signal whereall of R-, G-, and B-components are complete for each pixel.Specifically, the R- and B-component interpolation filter shown in FIG.5A is used for R- and B-components, and the G-component interpolationfilter shown in FIG. 5B is used for the G-component. Furthermore, in theG-component interpolation filter, it is required that an operation inthe x-direction has priority over an operation in the y-direction.

[0106] For example, when R-components are interpolated, onlyR-components in the Bayer array shown in FIG. 14 are input to the R- andB-component interpolation filter shown in FIG. 5A, and zero is input forB- and G-components to the R- and B-component interpolation filter. As aresult, an output from the R- and B-component interpolation filterresults in an interpolated R-component image. In the R- and B-componentinterpolation filter shown in FIG. 5A, the order of arithmeticoperations is not important because a weighting coefficient is the samefor both the x- and y-directions. However, in the G-componentinterpolation filter shown in FIG. 5B, an operation in the x-directionmust be performed prior to an operation in the y-direction because aweighting coefficient for the x-direction is different from that for they-direction. If an operation in the y-direction must be performed priorto an operation in the x-direction, weighting coefficients for the x-and y-directions shown in the G-interpolation filter of FIG. 5B arereplaced with each other. Each of the interpolation filters shown inFIGS. 5A and 5B is a low-path filter (which includes only positivecoefficients). Each of these filters has a size of an even-number ofpixels along both x- and y-directions. Furthermore, in each of thesefilters, coefficients of terms are symmetrically arranged with respectto both x- and y-directions of the filter. With such a symmetricalarrangement of coefficients, an effect of filtering can be uniformlyobtained, and as a result, an image can be faithfully reproduced.

[0107] In the second RGB interpolation section 12, filters having a sizeof an even-number of pixels are employed because at the adder 8 wheremiddle-high range luminance component YH is added to a low-frequencyluminance signal YL, a position of image data from the section 15 mustconform to that of image data from the multiplier 7. Specifically, asdescribed above, since the YH extraction filter of the middle-high rangeluminance component extraction section 6 must have a size of aneven-number of pixels whereas the interpolation filter used in the firstRGB interpolation section 4 has a size of an odd-number of pixels, imagedata of the extracted middle-high range luminance component YH reside inpositions between pixels. Therefore, new pixels must be created onboundaries between pixels in pixel data of the low-frequency luminancesignal YL, and to this end, a filter having a size of an even-number ofpixels is used in the second RGB interpolation section 12. In such anarrangement, pixel data of the low-frequency luminance signal YL ispositioned between pixels so as to conform to a position of image dataof the extracted middle-high range luminance component YH.

[0108] The white balance adjustment section 13 adjusts the white balancewith respect to the interpolated R-, G-, or B-components in each pixelaccording to the color temperature of light so as to correct the colorof an image.

[0109] The gamma correction section 14 processes the R-, G-, andB-components obtained after the white balance has been adjusted suchthat the R-, G-, and B-components conform to properties of a display orprinter for outputting an image.

[0110] The low-frequency luminance signal generation section 15generates a low-frequency luminance signal YL based on above expression(6) after gamma correction.

[0111] The color-difference signal generation section 16 generatescolor-difference signals Cr and Cb based on expressions (7) after gammacorrection:

Cr=0.70R−0.59G−0.11B Cb=−0.30R−0.59G+0.89B  (7)

[0112] The median filter 17 removes noise from the abovecolor-difference signals Cr and Cb. The size of the median filter 17depends on the quality of a CCD. That is, for example, a median filterhaving a size of 5 pixels ×5 pixels is used for a CCD which causes muchnoise, and a median filter having a size of 3 pixels×3 pixels is usedfor a CCD which causes small noise.

[0113] Hereinafter, a principle of the present invention will bedescribed in detail.

[0114] As described above, when a high-range luminance component isemphasized by a compensation filter having a frequency characteristicrepresented by graph c shown in FIG. 21, frequency components having afrequency higher than ½ of a sampling frequency fs are undesirablyemphasized as well, whereby jaggy is caused. In order to remove suchjaggy, according to the present invention, a middle-high range componentrepresented by graph e of FIG. 21 is mainly compensated for, and middle-and high-range components of an image are supplementarily compensatedfor. As shown by graph e of FIG. 21, the middle-high range component hasa maximum value of its amplitude at a position where the angularfrequency ω is between π/2 to π, and the amplitude thereof is zero whenthe angular frequency ω is π.

[0115] For the purpose of simplifying the description, a one-dimensionalfilter is used instead of a two-dimensional filter in an exampleillustrated below. A filter which can extract the middle-high luminancecomponent is a filter (A) shown in FIG. 6. The transfer function of thefilter (A) of FIG. 6 having a size of an even-number of pixels is asshown in expression (8). It should be noted that a center of the filter(A) of FIG. 6 having a size of an even-number of pixels is present at anintermediate position between pixels, and a position of a filter outputis also present at an intermediate position between pixels. The reasonthat a filter having a size of an even-number of pixels is used is toobtain a frequency characteristic represented by graph e of FIG. 21. Inthe filter (A) of FIG. 6, the number shown in each box represents a“weight” of a pixel corresponding to the box. $\begin{matrix}{\begin{matrix}{{H(z)} = {{- Z^{- 1.5}} + Z^{- 0.5} + Z^{0.5} - Z^{1.5}}} \\{= {{2{\cos \left( {0.5\omega} \right)}} - {2{\cos \left( {1.5\omega} \right)}}}}\end{matrix}{{{where}\quad z} = {e^{j\quad \omega} = {{\cos \quad \omega} + {j\quad \sin \quad \omega}}}}} & (8)\end{matrix}$

[0116] For the purpose of comparing the filter (A) with a middle-rangeluminance component compensation filter and a high-range luminancecomponent compensation filter which are used in a conventionaltechnique, the two-dimensional second-derivative filter of FIG. 17B isconverted into a one-dimensional filter to prepare a filter (B) of FIG.6, and the two-dimensional second-derivative filter of FIG. 17A isconverted into a one-dimensional filter to prepare a filter (C) of FIG.6. The transfer function of the filter (B) of FIG. 6 is as shown inexpression (9), and the transfer function of the filter (C) of FIG. 6 isas shown in expression (10): $\begin{matrix}\begin{matrix}{{H(z)} = {{- Z^{- 1}} + {2Z^{0}} - Z^{1}}} \\{= {2 - {2{\cos (\omega)}}}}\end{matrix} & (9) \\\begin{matrix}{{H(z)} = {{- Z^{- 2}} + {2Z^{0}} - Z^{2}}} \\{= {2 - {2{\cos \left( {2\omega} \right)}}}}\end{matrix} & (10)\end{matrix}$

[0117] The transfer functions represented by expressions (8), (9), and(10) are normalized, and frequency characteristics thereof are shown ingraphs A, B, and C, respectively, in FIG. 7. The frequencycharacteristic of expression (8) shown in graph A exhibits a maximumvalue of the amplitude when the angular frequency ω is 0.6π, andexhibits an amplitude of zero when the angular frequency ω is π. Thus,the compensation with middle-high luminance component can be achieved.

[0118] In the case of using a filter having a size of an odd-number ofpixels where a value (amplitude) of its transfer function is zero whenthe angular frequency ω is π, the amplitude thereof reaches a maximumvalue only when the angular frequency ω is π/2 (equivalent to fs/4). Forexample, referring to FIG. 8 which shows frequency characteristics ofthe filters (A), (D), and (E) of FIG. 6, the amplitude of the frequencycharacteristic for each of the filters (D) and (E), i.e., a filterhaving a size of an odd-number of pixels, is not zero when the angularfrequency ω is π. Thus, a filter having a size of an odd-number ofpixels causes noise or jaggy in a reproduced image as in a conventionaltechnique.

[0119] Thus, for the purpose of compensating for a high-range componentwithout emphasizing noise or jaggy, it is necessary to use a filterhaving a size of an even-number of pixels, such as the filter (A) ofFIG. 6. A filter (F) of FIG. 6 has a size of an even-number of pixelswhich is obtained by adding a high-order derivative term to the filter(A). The transfer function of the filter (F) is as shown in expression(11): $\begin{matrix}\begin{matrix}{{H(z)} = {Z^{- 2.5} - {5Z^{- 1.5}} + {4Z^{- 0.5}} + {4Z^{0.5}} - {5Z^{1.5}} + Z^{2.5}}} \\{= {{8{\cos \left( {0.5\omega} \right)}} - {10{\cos \left( {1.5\omega} \right)}} + {2{\cos \left( {2.5\omega} \right)}}}}\end{matrix} & (11)\end{matrix}$

[0120] Referring to FIG. 9, the frequency characteristic of a normalizedtransfer function (8) (filter (A) of FIG. 6) is represented by graph A,and the frequency characteristic of a normalized transfer function (11)(filter (F) of FIG. 6) is represented by graph F. Graph A exhibits apeak of amplitude at a position where ω=0.6π, and graph F exhibits apeak of amplitude at a position where ω=0.687π. In both of graph A andgraph F, the amplitude is zero at a position where ω=π. Thus, both ofthe filter (A) and filter (F) can be used to compensate for amiddle-high range luminance component. However, the filter (F) is moreuseful than the filter (A) because, when graph A (filter (A)) and graphF (filter (F)) exhibit the maximum amplitude, the angular frequency ωfor graph F is larger than that for graph A. In other words, by using afilter which exhibits the amplitude of zero at a position where ω=π,frequency components having a frequency higher than fs/2 can beprevented from causing noise. Furthermore, within a angular frequencyrange of π/2 to π, as a value of angular frequency (oat which thefrequency characteristic exhibits a maximum value of amplitude becomescloser to π, a higher-range luminance component can be compensated for.

[0121] Furthermore, using a filter having a size of an even-number ofpixels means that new pixels are created, not at original pixelpositions of a CCD, but at positions on borders between original pixels.Thus, a two-dimensionally arranged filter has an arrangement shown inFIG. 10. In FIG. 10, symbols “⊚” denote original pixel positions of theCCD, and symbols “◯” denote pixel data positions after a filter having asize of an even-number of pixels has been applied.

[0122] For the purpose of examining an operation of an image processingapparatus of the present invention, the frequency characteristic as toluminance signal processing when a one-dimensional filter is used is nowdescribed. Referring to FIG. 1, the description is made while referringto a series of processing shown in FIG. 1 since the primary-color CCDarea sensor 3 outputs image data until a luminance signal (YL+αYH) isobtained by the adder 8. Each process in the luminance generationsection 5, the white balance adjustment section 13, the gamma correctionsection 14, and the low-frequency luminance signal generation section 15does not influence a frequency distribution of image data, andtherefore, descriptions thereof are herein omitted.

[0123] In the first RGB interpolation section 4, R- and B-components areinterpolated by using a bilinear method of expressions (2) through (4),and a G-component is interpolated by using a median method of expression(5). With such interpolation processing, a middle-high range componentis attenuated, but is not completely removed.

[0124] The middle-high range luminance component extraction section 6uses the YH interpolation filters shown in FIGS. 2A and 2B to extract amiddle-high range luminance component YH. The extracted middle-highrange luminance component YH can be freely adjusted by using gain α. Itshould be noted that, for the purpose of simplifying the description, aloss of the middle-high range luminance component YH in the first RGBinterpolation section 4 is not considered.

[0125] The two-dimensional filters for extracting a middle-high rangeluminance component (first and second YH extraction filters F1 and F2shown in FIGS. 2A and 2B) correspond to the one-dimensional filter (F)of FIG. 6, the transfer function of the filter (F) is represented byexpression (11).

[0126] The second RGB interpolation section 12 uses, as a RGBinterpolation filter, two-dimensional filters shown in FIGS. 5A and 5B.The two-dimensional filters shown in FIGS. 5A and 5B correspond to aone-dimensional filter (G) of FIG. 6, the transfer function of thefilter (G) is represented by expression (12):

H(jω)=Z ^(−1.5)+3 Z ^(−0.5)+3 Z ^(0.5) +Z ^(1.5)  (12)

[0127] In FIG. 11, the frequency characteristic of expression (11) ofthe YH extraction filter is represented by graph J, and the frequencycharacteristic of expression (12) of the RGB interpolation filter whichis used in the second RGB interpolation section 12 is represented bygraph I. (It should be noted that FIG. 11 shows the normalizedfunctions.) The frequency characteristic of a luminance signal (YL+YH)is represented by graph H. From FIG. 11, it is clearly understood thatthe middle-high range luminance component YH has been compensated for.

[0128] Compensation of a middle-range luminance component and ahigh-range luminance component is supplementarily carried out by usingthe two filters shown in FIGS. 17A and 17B. For comparison, FIG. 12shows again the frequency characteristic of transfer function (11) ofthe YH extraction filters shown in FIGS. 2A and 2B which is in the formof a one-dimensional function, and the frequency characteristics oftransfer functions (9) and (10) of the two two-dimensionalsecond-derivative filters shown in FIGS. 17A and 17B which are in theform of a one-dimensional function. In FIG. 12, graph J shows thefrequency characteristic of filter (F) of FIG. 6 which is represented byexpression (11), graph B shows the frequency characteristic of thehigh-range luminance component extraction filter shown in FIG. 17B, andgraph C shows the frequency characteristic of the middle-range luminancecomponent extraction filter shown in FIG. 17A. Considering influence ofthe optical low-pass filter 2 (FIG. 20), a middle-range luminancecomponent and a high-range luminance component in graph H show in FIG.11 can be compensated for by utilizing the frequency characteristics ofgraph B and graph C show in FIG. 12. With only compensation withmiddle-high range component YH, a frequency characteristic cannot becompensated so as to be identical with the ideal characteristicrepresented by graph C of FIG. 20. However, by adjusting parameters 1and m of the composite contour emphasizing filter shown in FIG. 4, gaina and ENH gain β, a frequency characteristic which is closer to theideal characteristic represented by graph C can be obtained.

[0129] As described hereinabove, according to this embodiment of thepresent invention, as shown in FIG. 1, an image processing systemincludes a middle-high range luminance component YH generation routeincluding the first RGB interpolation section 4 and the middle-highrange luminance component extraction section 6 in parallel with alow-frequency luminance signal YL generation route including the secondRGB interpolation section 12, and the middle-high range luminancecomponent YH is added by the adder 8 to low-frequency luminance signalYL, whereby a middle-high range luminance component is compensated forby the middle-high range luminance component YH. Furthermore, amiddle-range luminance component and a high-range luminance componentare supplementarily compensated for by using a conventional compensationmethod. Thus, an image with higher resolution can be obtained ascompared with an image obtained by a conventional image processingsystem, while noise and jaggy can be prevented from appearing in areproduced image.

[0130] According to this embodiment of the present invention, in orderto prevent occurrence of false color, a color-difference signal isgenerated from a second RGB image signal which is generated byinterpolation with a low-pass filter in the second RGB interpolationsection 12 (FIG. 1), and then is subjected to the median filter 17(FIG. 1) before it is externally output. On the other hand, in order toobtain a desirable luminance signal, the middle-high range luminancecomponent YH is added by the adder 8 to low-frequency luminance signalYL, and a middle-range luminance component and high-range luminancecomponent in a resultant signal are supplementarily compensated. Whenthe middle-high range luminance component YH is added to low-frequencyluminance signal YL in the adder 8, the position of image data has to beadjusted. As described above, in the middle-high range luminancecomponent extraction section 6, for the purpose of extractingmiddle-high range luminance component YH, an extraction filter used mustbe a filter having a size of an even-number of pixels. Accordingly, thefirst RGB interpolation section 4 has to use, as a RGB interpolationfilter, a filter having a size of an odd-number of pixels, especially afilter having a size of 3 pixels×3 pixels, which is the best size inpreventing deterioration of middle-high range luminance component YH.Furthermore, the second RGB interpolation section 12 has to use, as aRGB interpolation filter, a filter having a size of an even-number ofpixels.

[0131] In the present embodiment, the color filter provided to the CCDarea sensor is a color filter whose RGB color arrangement based on theBayer array. However, the RGB color arrangement which can be used in thepresent invention is not limited to the Bayer array, but any RGB colorarrangement may be used.

[0132] The above descriptions as to an image processing apparatus of thepresent invention is now summarized. Referring to FIG. 1, the imageprocessing apparatus 1 processes RGB image data generated by the opticallow-pass filter 2 and the area sensor 3. In this image processing, alow-frequency luminance signal generated from the RGB image data has amiddle-high range luminance component which is attenuated due to theoptical low-pass filter 2 as compared with an ideal frequencycharacteristic which should be obtained by the optical low-pass filter2. The image processing apparatus I includes means of compensating forthe attenuated middle-high range luminance component of thelow-frequency luminance signal. Owing to such compensation means, asharper image can be obtained by contour emphasizing processing whilepreventing occurrence of noise and jaggy which may be caused during thecontour emphasizing processing.

[0133] According to the present invention, a middle-high range luminancecomponent of a low-frequency luminance signal which is attenuated ascompared with an ideal frequency characteristic in a range of apredetermined frequency (a ½ of sampling frequency fs) or smaller iscompensated. Thus, a sharp image can be obtained by contour emphasizingprocessing while preventing occurrence of noise and jaggy which may becaused when obtaining the sharp image.

[0134] According to the present invention, an image with high resolutioncan be obtained by contour emphasizing processing while preventingoccurrence of noise and jaggy which may be caused when obtaining thesharp image.

[0135] According to the present invention, a middle-high range componentwhose amplitude is zero when the angular frequency ω is π and has amaximum value at a position where the angular frequency ω is between π/2to π can be readily extracted.

[0136] According to the present invention, a uniform filtering effectcan be obtained, and as a result, an image can be faithfully reproduced.

[0137] According to the present invention, arithmetic operations in x-and y-directions of image data can be separately performed, andaccordingly, increases in the amount of arithmetic operations can besuppressed. Thus, YH extraction filtering can be readily implemented byhardware.

[0138] According to the present invention, a middle-high range luminancecomponent can be extracted while most-effectively preventingdeterioration of the middle-high range luminance component.

[0139] According to the present invention, when compensating for amiddle-high range luminance component of a low-frequency luminancesignal, a center of the middle-high range luminance component is presentat a position between neighboring pixels, and a center of thelow-frequency luminance signal is also present at a position betweenneighboring pixels. Thus, occurrence of a ghost in a reproduced imagecan be prevented.

[0140] According to the present invention, a G-component is interpolatedby using a median method, whereby attenuation of a high-range luminancecomponent is suppressed to a minimum level. R- and B-components areinterpolated by using a bilinear method, whereby noise is reduced. Thus,a contour of an image is emphasized, and the quality of the image can beimproved.

[0141] According to the present invention, by changing a ratio between amiddle-range luminance component and a high-range luminance component,the three-dimensional appearance (stereoscopic effect or stereophoniceffect) of an image can be adjusted according to user's preference.

[0142] According to the present invention, a middle/high-range componentextraction section can be readily formed of a single filter.

[0143] According to the present invention, the amount of noise includedin a color-difference signal varies according to, for example, thequality of an image capturing element such as a CCD. Thus, by selectingan appropriate median filter according to the amount of noise, acolor-difference signal with reduced noise can be generated.

[0144] Various other modifications will be apparent to and can bereadily made by those skilled in the art without departing from thescope and spirit of this invention. Accordingly, it is not intended thatthe scope of the claims appended hereto be limited to the description asset forth herein, but rather that the claims be broadly construed.

What is claimed is:
 1. An image processing apparatus for processing RGBimage data output from an image capturing element including aprimary-color filter, comprising: a middle-high range luminancecomponent compensation section for compensating for a middle-high rangeluminance component of a low-frequency luminance signal generated basedon the RGB image data such that the low-frequency luminance signal hassubstantially an ideal frequency luminance characteristic which is lowerthan or equal to a predetermined frequency.
 2. An image processingapparatus for processing RGB image data output from an image capturingelement including a primary-color filter, comprising: a middle-highrange luminance component extraction section for extracting amiddle-high range luminance component which has a zero amplitude at anangular frequency ω=π and a maximum amplitude at an angular frequency ωbetween π/2 and π from a first luminance signal generated based on RGBimage data; and a first synthesis section for adding the middle-highrange luminance component to a low-frequency luminance signal generatedbased on the RGB image data so as to generate a second luminance signal.3. An image processing apparatus according to claim 2, wherein themiddle-high range luminance component extraction section uses at leastone filter having a size of an even-number of pixels to arithmeticallyprocess the first luminance signal.
 4. An image processing apparatusaccording to claim 31 wherein the filter having a size of an even-numberof pixels is a two-dimensional filter and has coefficients symmetricallyarranged with respect to a x-direction and a y-direction.
 5. An imageprocessing apparatus according to claim 4, wherein: the filter having asize of an even-number of pixels includes a first low-pass filter havinga differentiation capability and a second low-pass filter; and adifference between an output obtained by arithmetically processing thefirst luminance signal using the first low-pass filter and an outputobtained by arithmetically processing the first luminance signal usingthe second low-pass filter is output as the middle-high range luminancecomponent.
 6. An image processing apparatus according to claim 5,further comprising: a first interpolation section for interpolatingmissing components among R-, G-, and B-components for each pixel beforethe generation of the first luminance signal, wherein the firstinterpolation section interpolates missing components by arithmeticallyprocessing the RGB image data using a filter having a size of 3 pixels×3pixels.
 7. An image processing apparatus according to claim 3, wherein:the filter having a size of an even-number of pixels includes a firstlow-pass filter having a differentiation capability and a secondlow-pass filter; and a difference between an output obtained byarithmetically processing the first luminance signal using the firstlow-pass filter and an output obtained by arithmetically processing thefirst luminance signal using the second low-pass filter is output as themiddle-high range luminance component.
 8. An image processing apparatusaccording to claim 7, further comprising: a first interpolation sectionfor interpolating missing components among R-, G-, and B-components foreach pixel before the generation of the first luminance signal, whereinthe first interpolation section interpolates missing components byarithmetically processing the RGB image data using a filter having asize of 3 pixels×3 pixels.
 9. An image processing apparatus according toclaim 8, further comprising: a second interpolation section forinterpolating missing components among R-, G-, and B-components for eachpixel before the generation of the low-frequency luminance signal,wherein the second interpolation section interpolates missing componentsby arithmetically processing the RGB image data using a filter having asize of an even-number of pixels.
 10. An image processing apparatusaccording to claim 9, wherein at least one of the first and secondinterpolation sections interpolates the RGB image data by using a medianmethod for a G-component and a bilinear method for R- and B-components.11. An image processing apparatus according to claim 9, furthercomprising: a median filtering section for removing, with a medianfilter, noise inherent to the image capturing element which is containedin a color-difference signal generated based on a RGB image signal fromthe second interpolation section; wherein the median filtering sectionchanges the size of the median filter according to an amount of thenoise.
 12. An image processing apparatus according to claim 3, furthercomprising: a first interpolation section for interpolating missingcomponents among R-, G-, and B-components for each pixel before thegeneration of the first luminance signal, wherein the firstinterpolation section interpolates missing components by arithmeticallyprocessing the RGB image data using a filter having a size of 3 pixels×3pixels.
 13. An image processing apparatus according to claim 2, furthercomprising: a first interpolation section for interpolating missingcomponents among R-, G-, and B-components for each pixel before thegeneration of the first luminance signal, wherein the firstinterpolation section interpolates missing components by arithmeticallyprocessing the RGB image data using a filter having a size of 3 pixels×3pixels.
 14. An image processing apparatus according to claim 13, furthercomprising: a second interpolation section for interpolating missingcomponents among R-, G-, and B-components for each pixel before thegeneration of the low-frequency luminance signal, wherein the secondinterpolation section interpolates missing components by arithmeticallyprocessing the RGB image data using a filter having a size of aneven-number of pixels.
 15. An image processing apparatus according toclaim 14, wherein at least one of the first and second interpolationsections interpolates the RGB image data by using a median method for aG-component and a bilinear method for R- and B-components.
 16. An imageprocessing apparatus according to claim 14, further comprising: a medianfiltering section for removing, with a median filter, noise inherent tothe image capturing element which is contained in a color-differencesignal generated based on a RGB image signal from the secondinterpolation section; wherein the median filtering section changes thesize of the median filter according to an amount of the noise.
 17. Animage processing apparatus according to claim 2, further comprising: asecond interpolation section for interpolating missing components amongR-, G-, and B-components for each pixel before the generation of thelow-frequency luminance signal, wherein the second interpolation sectioninterpolates missing components by arithmetically processing the RGBimage data using a filter having a size of an even-number of pixels. 18.An image processing apparatus according to claim 17, further comprising:a median filtering section for removing, with a median filter, noiseinherent to the image capturing element which is contained in acolor-difference signal generated based on a RGB image signal from thesecond interpolation section; wherein the median filtering sectionchanges the size of the median filter according to an amount of thenoise.
 19. An image processing apparatus according to claim 2, furthercomprising: a middle/high-range luminance component extraction sectionfor extracting at least one of a middle-range luminance component and ahigh-range luminance component based on the second luminance signal; anda second synthesis section for adding at least one of the middle-rangeluminance component and the high-range luminance component to the secondluminance signal so as to generate a third luminance signal.
 20. Animage processing apparatus according to claim 19, wherein themiddle/high-range luminance component extraction section arithmeticallyprocesses the second luminance signal by using one filter which has anadjustable coefficient.